SSZ-QL: calculate generalized indices for elements #15873

fernantho · 2025-10-15T15:29:59Z

What type of PR is this?
Feature

Which issues(s) does this PR fix?
Partially #15598

What does this PR do? Why is it needed?
This PR develop a way to calculate the Generalized Indices of a given path within a SSZ Object. To do so, it follows Consensus Spec's Merkle proofs.

A conversion from PathElement to Generalized Index is necessary to work with fastssz proofs library.

The code implemented walks the path within the SSZInfo struct in a consensus layer spec way. We have to take into account that the "path" input is slightly different:

def get_generalized_index(typ: SSZType, *path: PyUnion[int, SSZVariableName]) -> GeneralizedIndex:
    """
    Converts a path (eg. `[7, "foo", 3]` for `x[7].foo[3]`, `[12, "bar", "__len__"]` for
    `len(x[12].bar)`) into the generalized index representing its position in the Merkle tree.
    """

// GetGeneralizedIndexFromPath calculates the generalized index for a given path.
// To calculate the generalized index, two inputs are needed:
// 1. The sszInfo of the root info, to be able to navigate the SSZ structure
// 2. The path to the field (e.g., "field_a.field_b[3].field_c")
// It walks the path step by step, updating the generalized index at each step.
func GetGeneralizedIndexFromPath(info *sszInfo, path []PathElement) (uint64, error) {

The pythonic version expects a Path input [FieldA,"FieldB",3] while the Go version expects field_a.field_b[3].

Other notes for review

At the beginning, I considered implementing an implementation using recursion but this approach was discarded because of the inputs, as we already have the PathElement array, we can just loop this array.
Input format should be snake case as the ssz generated type names are in snake case.
Generalized Indices computed in tests were gotten from a Python script that relies on the Consensus Layer spec: https://github.com/fernantho/generalized-indices-ground-truth/blob/main/generalized_indices.ipynb
Multidimensional arrays GI will be handled at a later iteration.

Acknowledgements

I have read CONTRIBUTING.md.
I have included a uniquely named changelog fragment file.
I have added a description to this PR with sufficient context for reviewers to understand this PR.

… no recursion. Extended test coverage for bitlist and bitvectors. vectors need more testing

…he beginning. Swap to regex to gain flexibility.

…nction

encoding/ssz/query/generalized_index.go

…pe (uint64 instead of uint8)

…es.go

removed extractFieldName func.

…against vector.length/list.limit

encoding/ssz/query/path.go

encoding/ssz/query/path_test.go

encoding/ssz/query/path.go

encoding/ssz/query/generalized_index.go

encoding/ssz/query/generalized_index_test.go

Co-authored-by: Jun Song <[email protected]>

…e with length >= 1 If s does not contain sep and sep is not empty, Split returns a slice of length 1 whose only element is s.

…needed in extractFieldName function

syjn99

Overall good, my unofficial approval 👍

encoding/ssz/query/path_test.go

encoding/ssz/query/generalized_index_test.go

- renamed itemLengthFromInfo to itemLength (same name is in spec). - arranged all SSZ helpers.

rkapka · 2025-10-23T12:10:46Z

encoding/ssz/query/generalized_index.go

+	}
+
+	// Starting from the root generalized index
+	root := uint64(1)


Can you rename it to something like currentIndex? It's a bit odd to call it root since it's not a root of anything

Sure!
I got the inspiration for the root name from spec:

def get_generalized_index(typ: SSZType, *path: PyUnion[int, SSZVariableName]) -> GeneralizedIndex: """ Converts a path (eg. `[7, "foo", 3]` for `x[7].foo[3]`, `[12, "bar", "__len__"]` for `len(x[12].bar)`) into the generalized index representing its position in the Merkle tree. """ root = GeneralizedIndex(1) (...)

But I do not like it because I do not associate it to an index.

rkapka · 2025-10-23T12:25:26Z

encoding/ssz/query/path.go

+// e.g. "array[0][1]" -> []uint64{0, 1}. Errors if none are found or if any index is invalid.
+func extractArrayIndices(name string) ([]uint64, error) {
+	// Match all bracketed content, then we'll parse as unsigned to catch negatives explicitly
+	re := arrayIndexRegex


What's the purpose of assigning to a new variable instead of using arrayIndexRegex directly?

No purpose at all. I'll remove it.

rkapka · 2025-10-23T12:27:52Z

encoding/ssz/query/path.go

+		if strings.HasPrefix(raw, "-") {
+			return nil, fmt.Errorf("cannot process negative indices %q", raw)
+		}
+		idx, err := strconv.ParseUint(raw, 10, 64)
+		if err != nil {
+			return nil, fmt.Errorf("invalid array index: %w", err)
+		}


Looking at the documentation for ParseUint it seems that it doesn't allow negative numbers, so the HasPrefix check is unnecessary

// ParseUint is like [ParseInt] but for unsigned numbers.
//
// A sign prefix is not permitted.

Thanks. Totally unnecessary check.

rkapka · 2025-10-23T12:33:56Z

encoding/ssz/query/path.go

-			if len(parts) != 2 {
-				return nil, fmt.Errorf("invalid index notation in path element %s", elem)
-			}
+		re := regexp.MustCompile(`^\s*len\s*\(\s*([^)]+?)\s*\)\s*$`)


How about extracting this regex to a package-level variable, just like you did with arrayIndexRegex?

I forgot to extract this one. It's now at the package-level variable.

rkapka · 2025-10-23T12:34:18Z

encoding/ssz/query/path.go

-			}
+		re := regexp.MustCompile(`^\s*len\s*\(\s*([^)]+?)\s*\)\s*$`)
+		matches := re.FindStringSubmatch(processingField)
+		if len(matches) == 2 {


It would be nice to add an explanation why 2 is expected

Added explanation. I'm now considering regular expressions an overkill, when I added them I was considering input validation and correction.

encoding/ssz/query/path.go

rkapka · 2025-10-23T12:42:29Z

encoding/ssz/query/path_test.go

 			},
 			wantErr: false,
 		},
+		{


Can you add even more test cases? The ones I can think of:

leading double dot --> error

trailing dot --> error

len(data) --> error

len(data.target.root) --> ok

len(data.target.root).foo --> error

data.target.len(root) --> error

The easiest way to get a bunch of test cases it to pass the regex to an AI and ask it to generate them. I expect it will overdo this, but some cases can be useful.

Adding them, but I see we do not follow the same convention for requests:

len(data.target.root) --> error

data.target.len(root) --> ok

As we firstly split the raw input by ., having this input len(data.target.root) would result in:

len(data

target

root)

leading to a wrong outcome.

On the contrary, this would succeed data.target.len(root)

data

target

len(root)

We must properly specify the input format for these queries.

len(data) --> error

I do not see an error here neither. Imagine we are querying validators field length in beacon state.
In this case the query would contain len(validators)

rkapka · 2025-10-23T12:47:58Z

encoding/ssz/query/generalized_index.go

+// Helpers for Generalized Index calculation per type
+
+// calculateLengthGeneralizedIndex calculates the generalized index for a length field.
+// note: length fields are only valid for List and Bitlist types. Multi-dimensional arrays are not supported.


Shouldn't this be supported for Vector and Bitvector too?

In relation to this, I also followed the spec algo:

for p in path: # If we descend to a basic type, the path cannot continue further assert not issubclass(typ, BasicValue) if p == "__len__": + assert issubclass(typ, (List, ByteList)) typ = uint64 root = GeneralizedIndex(root * 2 + 1)

To my understanding, there is no length field for Vector and Bitvector as they have fixed size determined by their type.

rkapka · 2025-10-23T12:49:20Z

encoding/ssz/query/generalized_index.go

+
+// calculateLengthGeneralizedIndex calculates the generalized index for a length field.
+// note: length fields are only valid for List and Bitlist types. Multi-dimensional arrays are not supported.
+func calculateLengthGeneralizedIndex(fieldSsz *SszInfo, element PathElement, root uint64) (*SszInfo, uint64, error) {


Similarly to the comment above (changing root to currentIndex), can you rename the root param to something like parentIndex (or something else more suitable)?

Re-renamed them 😅
changed all of them from root to currentIndex. But now I got to this comment and, for these functions, I like this parentIndex more than currentIndex.

encoding/ssz/query/generalized_index.go

…turns from calculate<Type>GeneralizedIndex functions

Co-authored-by: Radosław Kapka <[email protected]>

fernantho added 8 commits October 15, 2025 15:25

added tests for calculating generalized indices

50118d4

added first version of GI calculation walking the specified path with…

e77e465

… no recursion. Extended test coverage for bitlist and bitvectors. vectors need more testing

refactored code. Detached PathElement processing, currently done at t…

e00c804

…he beginning. Swap to regex to gain flexibility.

added an updateRoot function with the GI formula. more refactoring

83596c5

added changelog

787bb13

replaced TODO tag

253c1b6

udpated some comments

ed62201

simplified code - removed duplicated code in processingLengthField fu…

a2154e3

…nction

fernantho marked this pull request as ready for review October 17, 2025 12:56

fernantho added 2 commits October 17, 2025 14:56

Merge branch 'develop' into feat/ssz-ql-parse-path-to-generalized-index

62646ff

run gazelle

fe4d7fe

syjn99 reviewed Oct 18, 2025

View reviewed changes

fernantho added 10 commits October 20, 2025 11:40

merging all input path processing into path.go

96f1c4d

reviewed Jun's feedback

9e0314e

removed unnecessary idx pointer var + fixed error with length data ty…

a1de521

…pe (uint64 instead of uint8)

refactored path.go after merging path elements from generalized_indic…

24a1fff

…es.go

re-computed GIs for tests as VariableTestContainer added a new field.

43835e3

added minor comment - rawPath MUST be snake case

eb7637c

removed extractFieldName func.

fixed vector GI calculation - updated tests GIs

1baa32a

removed updateRoot function in favor of inline code

d5b1227

path input data enforced to be snake case

e9741e4

added sanity checks for accessing outbound element indices - checked …

73e3ee7

…against vector.length/list.limit

fernantho force-pushed the feat/ssz-ql-parse-path-to-generalized-index branch from ce17749 to 73e3ee7 Compare October 21, 2025 08:02

fernantho added 2 commits October 21, 2025 10:37

Merge branch 'develop' into feat/ssz-ql-parse-path-to-generalized-index

b65fff9

fixed issues triggered after merging develop

d800a18

syjn99 reviewed Oct 21, 2025

View reviewed changes

fernantho and others added 4 commits October 21, 2025 12:27

Removed redundant comment

b276b68

Co-authored-by: Jun Song <[email protected]>

removed unreachable condition as strings.Split always return a slic…

e0c8878

…e with length >= 1 If s does not contain sep and sep is not empty, Split returns a slice of length 1 whose only element is s.

added tests to cover edge cases + cleaned code (toLower is no longer …

9dfa152

…needed in extractFieldName function

added Jun's feedback + more testing

6478e00

fernantho added 2 commits October 23, 2025 08:51

removed toSnakeCase conversion.

7a808ca

moved isBasicType func to its natural place, SSZType

da278ed

syjn99 reviewed Oct 23, 2025

View reviewed changes

encoding/ssz/query/path_test.go Outdated Show resolved Hide resolved

encoding/ssz/query/generalized_index_test.go Outdated Show resolved Hide resolved

fernantho added 3 commits October 23, 2025 09:40

cosmetic refactor

f3f9a60

- renamed itemLengthFromInfo to itemLength (same name is in spec). - arranged all SSZ helpers.

cleaned tests

0b05112

Merge branch 'develop' into feat/ssz-ql-parse-path-to-generalized-index

6485eb7

rkapka reviewed Oct 23, 2025

View reviewed changes

fernantho and others added 7 commits October 23, 2025 18:30

renamed "root" to "index"

385650d

removed unnecessary check for negative integers. Replaced %q for %s.

83284c7

refactored regex variables and prevented re-assignation

900f971

added length regex explanation

2c8885b

added more testing for stressing regex for path processing

212d31b

renamed currentIndex to parentIndex for clarity and documented the re…

4aa7b82

…turns from calculate<Type>GeneralizedIndex functions

Update encoding/ssz/query/generalized_index.go

9d850e6

Co-authored-by: Radosław Kapka <[email protected]>

rkapka previously approved these changes Oct 27, 2025

View reviewed changes

Merge branch 'develop' into feat/ssz-ql-parse-path-to-generalized-index

30fc2a7

rkapka enabled auto-merge October 27, 2025 17:02

run gazelle

c61cdf7

auto-merge was automatically disabled October 27, 2025 17:10
Head branch was pushed to by a user without write access

fernantho dismissed rkapka’s stale review via c61cdf7 October 27, 2025 17:10

rkapka enabled auto-merge October 27, 2025 17:57

rkapka previously approved these changes Oct 27, 2025

View reviewed changes

fixed never asserted error. Updated error message

6107655

auto-merge was automatically disabled October 27, 2025 22:16
Head branch was pushed to by a user without write access

fernantho dismissed rkapka’s stale review via 6107655 October 27, 2025 22:16

rkapka approved these changes Oct 27, 2025

View reviewed changes

rkapka enabled auto-merge October 27, 2025 23:07

rkapka added this pull request to the merge queue Oct 27, 2025

Merged via the queue into OffchainLabs:develop with commit 10a2f06 Oct 27, 2025
22 checks passed

SSZ-QL: calculate generalized indices for elements #15873

SSZ-QL: calculate generalized indices for elements #15873

Uh oh!

Conversation

fernantho commented Oct 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

syjn99 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fernantho Oct 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

fernantho commented Oct 15, 2025 •

edited

Loading

fernantho Oct 24, 2025 •

edited

Loading